System and method for CPI load balancing in SMT processors

ABSTRACT

A system and method for scheduling threads in a Simultaneous Multithreading (SMT) processor environment utilizing multiple SMT processors is provided. Poor performing threads that are being run on each of the SMT processors are identified. After being identified, the poor performing threads are moved to a different SMT processor. Data is captured regarding the performance of threads. In one embodiment, this data includes each threads&#39; CPI value. When a thread is moved, data regarding the thread and its performance at the time it was moved is recorded along with a timestamp. The data regarding previous moves is used to determine whether a thread&#39;s performance is improved following the move.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates in general to a system and method forscheduling threads on SMT processors. More particularly, the presentinvention relates to a system and method that uses a measurement todetermine processing threads that are compatible with one another forSMT scheduling purposes.

2. Description of the Related Art

The fundamental structure of a modern computer includes peripheraldevices to communicate information to and from the outside world; suchperipheral devices may be keyboards, monitors, tape drives,communication lines coupled to a network, etc. Also included in thebasic structure of the computer is the hardware necessary to receive,process, and deliver this information from and to the outside world,including busses, memory units, input/output (I/O) controllers, storagedevices, and at least one central processing unit (CPU), etc. The CPU isthe brain of the system. It executes the instructions which comprise acomputer program and directs the operation of the other systemcomponents.

From the standpoint of the computer's hardware, most systems operate infundamentally the same manner. Processors actually perform very simpleoperations quickly, such as arithmetic, logical comparisons, andmovement of data from one location to another. Programs which direct acomputer to perform massive numbers of these simple operations give theillusion that the computer is doing something sophisticated. What isperceived by the user as a new or improved capability of a computersystem, however, may actually be the machine performing the same simpleoperations, but much faster. Therefore continuing improvements tocomputer systems require that these systems be made ever faster.

One measurement of the overall speed of a computer system, also calledthe throughput, is measured as the number of operations performed perunit of time. Conceptually, the simplest of all possible improvements tosystem speed is to increase the clock speeds of the various components,particularly the clock speed of the processor. If everything runs twiceas fast but otherwise works in exactly the same manner, the system willperform a given task in half the time. Computer processors which wereconstructed from discrete components years ago performed significantlyfaster by shrinking the size and reducing the number of components;eventually the entire processor was packaged as an integrated circuit ona single chip. The reduced size made it possible to increase the clockspeed of the processor, and accordingly increase system speed.

Despite the enormous improvement in speed obtained from integratedcircuitry, the demand for ever faster computer systems still exists.Hardware designers have been able to obtain still further improvementsin speed by greater integration, by further reducing the size of thecircuits, and by other techniques. Designers, however, think thatphysical size reductions cannot continue indefinitely and there arelimits to continually increasing processor clock speeds. Attention hastherefore been directed to other approaches for further improvements inoverall speed of the computer system.

Without changing the clock speed, it is still possible to improve systemspeed by using multiple processors. The modest cost of individualprocessors packaged on integrated circuit chips has made this practical.The use of slave processors considerably improves system speed byoff-loading work from the master processor to the slave processor. Forinstance, slave processors routinely execute repetitive and singlespecial purpose programs, such as input/output device communications andcontrol. It is also possible for multiple CPUs to be placed in a singlecomputer system, typically a host-based system which services multipleusers simultaneously. Each of the different CPUs can separately executea different task on behalf of a different user, thus increasing theoverall speed of the system to execute multiple tasks simultaneously.

It is more difficult, however, to improve the speed at which a singletask, such as an application program, executes. Coordinating theexecution and delivery of results of various functions among multipleCPUs is a tricky business. For slave I/O processors this is not sodifficult because the functions are pre-defined and limited but formultiple CPUs executing general purpose application programs it is muchmore difficult to coordinate functions because, in part, systemdesigners do not know the details of the programs in advance. Mostapplication programs follow a single path or flow of steps performed bythe processor. While it is sometimes possible to break up this singlepath into multiple parallel paths, a universal application for doing sois still being researched. Generally, breaking a lengthy task intosmaller tasks for parallel processing by multiple processors is done bya software engineer writing code on a case-by-case basis. This ad hocapproach is especially problematic for executing commercial transactionswhich are not necessarily repetitive or predictable.

Thus, while multiple processors improve overall system performance,there are still many reasons to improve the speed of the individual CPU.If the CPU clock speed is given, it is possible to further increase thespeed of the CPU, i.e., the number of operations executed per second, byincreasing the average number of operations executed per clock cycle. Acommon architecture for high performance, single-chip microprocessors isthe reduced instruction set computer (RISC) architecture characterizedby a small simplified set of frequently used instructions for rapidexecution, those simple operations performed quickly as mentionedearlier. As semiconductor technology has advanced, the goal of RISCarchitecture has been to develop processors capable of executing one ormore instructions on each clock cycle of the machine. Another approachto increase the average number of operations executed per clock cycle isto modify the hardware within the CPU. This throughput measure, clockcycles per instruction, is commonly used to characterize architecturesfor high performance processors. Instruction pipelining and cachememories are computer architectural features that have made thisachievement possible. Pipeline instruction execution allows subsequentinstructions to begin execution before previously issued instructionshave finished. Cache memories store frequently used and other datanearer the processor and allow instruction execution to continue, inmost cases, without waiting the full access time of a main memory. Someimprovement has also been demonstrated with multiple execution unitswith look ahead hardware for finding instructions to execute inparallel.

For both in-order and out-of-order completion of instructions insuperscalar systems, pipelines will stall under certain circumstances.An instruction that is dependent upon the results of a previouslydispatched instruction that has not yet completed may cause the pipelineto stall. For instance, instructions dependent on a load/storeinstruction in which the necessary data is not in the cache, i.e., acache miss, cannot be completed until the data becomes available in thecache. Maintaining the requisite data in the cache necessary forcontinued execution and to sustain a high hit ratio, i.e., the number ofrequests for data compared to the number of times the data was readilyavailable in the cache, is not trivial especially for computationsinvolving large data structures. A cache miss can cause the pipelines tostall for several cycles, and the total amount of memory latency will besevere if the data is not available most of the time. Although memorydevices used for main memory are becoming faster, the speed gap betweensuch memory chips and high-end processors is becoming increasinglylarger. Accordingly, a significant amount of execution time in currenthigh-end processor designs is spent waiting for resolution of cachemisses and these memory access delays use an increasing proportion ofprocessor execution time.

Another technique to improve the efficiency of hardware within the CPUis to divide a processing task into independently executable sequencesof instructions called threads. This technique is related to breaking alarger task into smaller tasks for independent execution by differentprocessors, except here the threads are to be executed by the sameprocessor. When a CPU then, for any of a number of reasons, cannotcontinue the processing or execution of one of these threads, the CPUswitches to and executes another thread. The term “multithreading” asdefined in the computer architecture community is not the same as thesoftware use of the term which means one task subdivided into multiplerelated threads. In the architecture definition, the threads may beindependent. Therefore “hardware multithreading” is often used todistinguish the two uses of the term.

Traditional forms of hardware multithreading involves replicating theprocessor registers for each thread. For instance, for a processorimplementing the architecture provided by the IBM Corporation under thetrade name PowerPC™ to perform multithreading, the processor mustmaintain N states to run N threads. Accordingly, the following arereplicated N times: general purpose registers, floating point registers,condition registers, floating point status and control register, countregister, link register, exception register, save/restore registers, andspecial purpose registers.

Additionally, the special buffers, such as a segment lookaside buffer,can be replicated or each entry can be tagged with the thread numberand, if not, must be flushed on every thread switch. Also, some branchprediction mechanisms, e.g., the correlation register and the returnstack, should also be replicated. Fortunately, there is no need toreplicate some of the larger functions of the processor such as: levelone instruction cache (L1 I-cache), level one data cache (L1 D-cache),instruction buffer, store queue, instruction dispatcher, functional orexecution units, pipelines, translation lookaside buffer (TLB), andbranch history table.

Simultaneous multithreading (SMT) is a technique that permits multipleindependent threads to issue multiple instructions each cycle to asuperscalar processor's functional units. SMT combines themultiple-instruction features of modern superscalar processors with thelatency-hiding ability of multithreaded architectures. Unlikeconventional multithreaded architectures, which depend on fast contextswitching to share processor execution resources, all hardware contextsin an SMT processor are active simultaneously, competing each cycle forall available resources. This dynamic sharing of the functional unitsallows simultaneous multithreading to substantially increase throughput,attacking the two major impediments to processor utilization—longlatencies and limited per-thread parallelism. Multiple SMT processorscan be included in a computer system allowing the computer to performsimultaneous multithreading on a plurality of computers.

A challenge, however, faced by computers with a plurality of SMTprocessors is that the software threads being executed by the SMTprocessor contend for some of the same processor-based resources, suchfunctional and execution units. As used herein, the term “thread” refersto a software thread, unless otherwise noted. If two threads are bothrepeatedly contending for the same processor-based resource, one threadwill wait (or gets swapped out) while the other thread uses theresource. Waiting for resources decreases overall system throughput aswell as the efficiency of the individual threads. In a multi-processorenvironment, a thread may not perform well on a given SMT processorbecause of the other threads that are running on that processor.However, traditional systems do not move threads from one SMT processorto another.

What is needed, therefore, is a system and method that identifiesthreads that are performing poorly on one SMT processor and move thepoor performing thread to another SMT processor. What is a furtherneeded is a system and method that swaps poor performing threads betweenSMT processors. Finally, what is needed is a system and method thatdetermines whether a thread's performance improves after it has beenmoved to another SMT processor.

SUMMARY

It has been discovered that the aforementioned challenges are addressedusing a system and method that identifies a poor performing threadrunning on one SMT processor and moves it to a different SMT processor.Performance data is recorded for software threads running on thecomputer system. In one embodiment, the performance data includes eachthreads' cycles-per-instruction (CPI) which is determined by dividingthe number of cycles that occurred in the processor by the number ofinstructions that were executed. The lower the CPI value, the better thethread is performing and, conversely, the higher the CPI value the worsethe thread is performing. In one embodiment the performance is averagedover a plurality of executions.

A poor performing thread (i.e., one with a high CPI value) is moved froma first run queue to a second run queue, with the first run queuecorresponding to its current SMT processor and the second run queuecorresponding to a different SMT processor. In one embodiment, poorperforming threads are swapped between processors. In this manner, athread that may perform poorly on one processor may perform adequatelyon a second processor. One reason that the performance may be improvedis that the other threads running on the newly assigned processor maynot need the same processor-based resources as the moved thread. Withless contention for the processor-based resources, the thread'sperformance is improved in the SMT processor environment.

When a thread completes execution (i.e., is preempted, time sliced,etc.), the thread's CPI during the last execution is recorded andchecked. If the thread is performing worst than a given threshold, acheck is made to see whether the thread has previously been moved and,if it has been moved, whether its performance has improved as a resultof being moved. If the thread's performance is worse after being movedor if the thread has not recently been moved then the thread isidentified as a potential thread to be moved. In one embodiment, onethread from each SMT processor's run queue is identified as poorperforming and moved to a different run queue. When a thread is moved, adata structure is updated capturing performance data regarding thethread and a timestamp in order to subsequently determine whether thethread's performance is improving. With systems with more than two SMTprocessors, poor performing threads can be moved from one processor tothe next in a variety of fashions. For example, a round-robinimplementation can be used to swap poor performing threads, orprocessors can be paired with one another with the poor performingthreads swapped between the paired processors.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations, and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. Otheraspects, inventive features, and advantages of the present invention, asdefined solely by the claims, will become apparent in the non-limitingdetailed description set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings. The use of the samereference symbols in different drawings indicates similar or identicalitems.

FIG. 1 is a high level diagram of a plurality of threads being scheduledfor concurrent execution on an SMT processor;

FIG. 2 is a diagram showing the scheduler using thread measurement dataand run queue data to schedule threads on an SMT processor;

FIG. 3 is a flowchart showing the steps taken by a scheduler indetermining a thread to dispatch onto an SMT processor;

FIG. 4 is a flowchart showing the steps taken to update a thread'scompatibility list;

FIG. 5 is a flowchart showing the steps taken to remove entries from athread's compatibility list;

FIG. 6 is a flowchart showing the steps taken to periodically clean upthe compatibility lists found in the thread control block;

FIG. 7 is a diagram showing the scheduler swapping a poor performingthread from one SMT processor to another SMT processor in order toimprove overall system performance;

FIG. 8 is a flowchart showing the steps taken to update a thread's CPI;

FIG. 9 is a flowchart showing the steps taken to swap poor performingthreads between SMT processors;

FIG. 10 is a flowchart showing the steps taken to identify poorperforming threads to swap in a multiple SMT processor system;

FIG. 11 is a flowchart showing the steps taken to swap poor performingthreads between SMT processors; and

FIG. 12 is a block diagram of an information handling system capable ofimplementing the present invention.

DETAILED DESCRIPTION

The following is intended to provide a detailed description of anexample of the invention and should not be taken to be limiting of theinvention itself. Rather, any number of variations may fall within thescope of the invention which is defined in the claims following thedescription.

FIG. 1 is a high level diagram of a plurality of threads being scheduledfor concurrent execution on an SMT processor. Scheduler 100 reads threaddata corresponding to a plurality of threads 110. In one embodiment, thetread data is stored in a thread control block (TCB) that is used by thesystem to maintain and manage the threads currently in existence.

Scheduler 100 dispatches threads to execute on processor 120 thatsupports simultaneous multithreading. Simultaneous multithreading (SMT)is a technique that permits multiple independent threads to issuemultiple instructions each cycle to a superscalar processor's functionalunits. SMT combines the multiple-instruction features of modernsuperscalar processors with the latency-hiding ability of multithreadedarchitectures. Unlike conventional multithreaded architectures, whichdepend on fast context switching to share processor execution resources,all hardware contexts in an SMT processor are active simultaneously,competing each cycle for all available resources. This dynamic sharingof the functional units allows simultaneous multithreading tosubstantially increase throughput, attacking the two major impedimentsto processor utilization long latencies and limited per-threadparallelism.

Processor threads 130 and 140 represent two threads that are executingsimultaneously on processor 120 and competing for processor resources150, 160, and 170. Depending on the tasks being performed, executablethreads 110 each have different needs for the processor resources. Forexample, some threads may be data intensive, needing extensive access tothe processor's “load/store” resource, while other threads may becomputationally intensive and require extensive access to theprocessor's “adder” resource or floating-point resource.

Because the resources are shared amongst the threads that are currentlyexecuting, if each thread needs to perform the same function, one of thethreads will have to wait while the other thread receives access to thefunction. If contention for resources is high between processes, thenthe processes will take more time to complete than if contention islower.

Data is maintained for each executable thread indicating each thread'scompatibility with other threads. For example, if two threads run welltogether (i.e., each have a low Cycles Per Instruction when runningtogether), this information will be used by the scheduler to preferablyhave both of these threads run at the same time.

FIG. 2 is a diagram showing the scheduler using thread measurement dataand run queue data to schedule threads on an SMT processor. When athread completes execution on SMT processor 250, scheduler 200dispatches the next thread for execution. If a thread from Run Queue A(270) completes, then another thread from Run Queue A is dispatched.Likewise, if a thread from Run Queue B (285) completes, then thescheduler dispatches another thread that is ready to execute from RunQueue B.

In order to determine which thread to dispatch, scheduler 200 determineswhich thread is currently running on the other processor thread. Forexample, if the thread that is ending is from Run Queue B, then thethread identifier corresponding to the thread that is currently runningin Processor Thread A (255) is retrieved. Likewise, if the thread thatis ending is from Run Queue A, then the thread identifier correspondingto the thread that is currently running in Processor Thread B (260) isretrieved. Run queues include data regarding the identifier of thethread (identifier 275 for threads in Run Queue A and identifier 290 forthreads in Run Queue B) as well as data indicating whether the thread isready to execute (indicator 280 for threads in Run Queue A and indicator295 for threads in Run Queue B).

The scheduler then determines if any “compatible” threads are ready toexecute. For example, if threads “aaa” from Run Queue A is running withthread “bbb” from Run Queue B and thread “bbb” is ending, then thescheduler determines if any threads that are compatible with thread“aaa” are ready to run. In one embodiment, the scheduler reads thecompatibility information for thread “aaa” from its entry in threadcontrol block 210. Thread control block 210 includes a number of datafields for each thread. This data includes the identifier of the thread215, fields for up to three identifiers corresponding to compatiblethreads (identifiers 220, 235, and 245), and the Cycles Per Instruction(CPI) that were achieved when the compatible threads were running withthe compatible threads (CPI 225 corresponding to thread ID 220, CPI 235corresponding to thread ID 230, and CPI 245 corresponding to thread ID240). In one embodiment, the CPI stored in fields 225, 235, and 245 isthe last CPI that occurred during the last execution of the thread withthe compatible thread. In another embodiment, an average CPI ismaintained for the various threads and the average CPI that has occurredduring executions of the thread with the compatible thread are stored inthe CPI fields.

In the example shown, thread “aaa” is most compatible with thread “bbb”as it has the lowest CPI (0.7) when running with thread “bbb.” Thread“aaa” is also compatible with threads “ddd” (CPI of 1.0) and “fff” (CPIof 1.2). The scheduler determines whether thread “ddd” is ready toexecute and, if so, dispatches the thread from Run Queue B to SMTProcessor 250 in the processor's “B” thread space (260). If thread “ddd”is not ready to execute, then the scheduler determines whether thread“fff” is ready to execute, and if so the scheduler dispatches it. Ifneither threads “ddd” or “fff” are ready to execute, the next threadthat is ready to run from Run Queue B is selected and dispatched by thescheduler.

In addition, when threads finish running the CPI is updated. Somecompatible thread IDs are “[null]” (an empty slot) which indicates thatnot enough compatible threads have been found to fill all the slots.When a thread finishes execution, the CPI of the thread is captured andcompared to a CPI threshold value 265. If the thread's CPI is betterthan the threshold amount, then the CPI measurement and identifier maybe placed into a compatibility slot if (1) an empty slot exists, or (2)no empty slots exist but the newly captured CPI is better than one ofthe current compatible threads (in which case the newly captured CPI andthread identifier replaces the current compatible thread with thehighest, i.e., poorest, CPI).

FIG. 3 is a flowchart showing the steps taken by a scheduler indetermining a thread to dispatch onto an SMT processor. Processingcommences at 300 whereupon, at step 305, the scheduler receives a noticethat a currently running thread is about to task out.

A determination is made as to which run queue includes the completingthread (decision 310). If the completing thread is on run queue “A”,then decision 310 branches to “yes” branch 318 in order to retrieve thenext thread to dispatch from run queue “A.” At step 320, thecompatibility list is checked to determine which threads on run queue“A” are compatible with the thread that is currently executing onprocessor thread “B” (which receives threads from run queue “B”). Thischeck is made by reading the compatibility data stored in thread controlblock 315 (for a more detailed example of the data stored in the threadcontrol block, see block 210 in FIG. 2).

A determination is made as to whether there are any compatible threadslisted for the thread running on processor thread “B” (decision 325). Ifthere are no compatible threads listed, then decision 325 branches to“no” branch 326 whereupon, at step 350, the next available thread fromrun queue “A” that is ready to run is dispatched and processing ends at395.

On the other hand, if one or more threads listed in the thread controlblock are compatible with the thread currently running on processorthread “B,” then decision 325 branches to “yes” branch 328 whereupon, atstep 330, the most compatible thread (i.e., the one with the lowest CPI)is checked by reading data from run queue “A” to determine if it isready to run. A determination is made as to whether the last checkedthread is ready to run (decision 335). If the last checked thread isready to run, decision 335 branches to “yes” branch 338 whereupon, atstep 340, the thread is dispatched. On the other hand, if the compatiblethread is not ready to run, decision 335 branches to “no” branch 342whereupon a determination is made as to whether there are any morecompatible threads listed in the thread control block (decision 345). Ifthere are more compatible threads listed, decision 345 branches to “yes”branch 346 which loops back to see if this thread is ready to run. Thislooping continues until either a compatible, ready to run thread isfound (decision 335 branching to “yes” branch 338), or there are no morecompatible threads to check. If there are no more compatible threads tocheck, decision 345 branches to “no” branch 348 whereupon, at step 350,the next available (ready to run) thread from run queue “A” isdispatched. Processing thereafter ends at 395.

Returning to decision 310, if the completing thread is on run queue“B,”, decision 310 branches to branch 352 whereupon a synonymous set ofdecisions and steps are performed to determine which thread from runqueue “B” should be dispatched. The details of these steps are asfollows:

At step 355, the compatibility list is checked to determine whichthreads on run queue “B” are compatible with the thread that iscurrently executing on processor thread “A” (which receives threads fromrun queue “A”). This check is made by reading the compatibility datastored in thread control block 315.

A determination is made as to whether there are any compatible threadslisted for the thread running on processor thread “A” (decision 360). Ifthere are no compatible threads listed, then decision 360 branches to“no” branch 362 whereupon, at step 390, the next available thread fromrun queue “B” that is ready to run is dispatched and processing ends at395.

On the other hand, if one or more threads listed in the thread controlblock are compatible with the thread currently running on processorthread “A,” then decision 360 branches to “yes” branch 364 whereupon, atstep 365, the most compatible thread (i.e., the one with the lowest CPI)is checked by reading data from run queue “B” to determine if it isready to run. A determination is made as to whether the last checkedthread is ready to run (decision 375). If the last checked thread isready to run, decision 375 branches to “yes” branch 378 whereupon, atstep 380, the thread is dispatched. On the other hand, if the compatiblethread is not ready to run, decision 375 branches to “no” branch 382whereupon a determination is made as to whether there are any morecompatible threads listed in the thread control block (decision 385). Ifthere are more compatible threads listed, decision 385 branches to “yes”branch 386 which loops back to see if this thread is ready to run. Thislooping continues until either a compatible, ready to run thread isfound (decision 375 branching to “yes” branch 378), or there are no morecompatible threads to check. If there are no more compatible threads tocheck, decision 385 branches to “no” branch 388 whereupon, at step 390,the next available (ready to run) thread from run queue “B” isdispatched. Processing thereafter ends at 395.

FIG. 4 is a flowchart showing the steps taken to update a thread'scompatibility list. Processing commences at 400 whereupon, at step 405,the thread identifier for the thread that just completed executing onone of the processor threads is retrieved along with the threadidentifier for the thread that is still executing on the other processorthread. Next, at step 410, the CPI that was achieved during the timethat the thread that just completed and the thread that is stillexecuting is retrieved.

A determination is made as to whether the retrieved CPI is less than orequal to (i.e., better than) the compatibility threshold that wasestablished (decision 415). If the CPI is greater than the thresholdvalue, then the threads are not considered to be “compatible.” Thethreshold value is a tunable value. The higher the value, the morethreads will be considered “compatible” yet, because of the highervalue, the CPIs will not necessarily greatly improve overall systemperformance. On the other hand, lowering the threshold value more likelyensures that “compatible” threads, when available, will perform welltogether, yet because of the lower threshold value fewer compatiblethreads may be identified. Thus, tuning the compatibility threshold maybe necessary depending upon the type of processing being performed by agiven computer system.

If the CPI is greater than the threshold value, decision 415 branches to“no” branch 418 whereupon the threads are judged as being “notcompatible” and any entries indicating that the threads are compatibleare removed (predefined process 420, see FIG. 5 for processing details),and processing ends at 425.

On the other hand, if the CPI is less than or equal to the thresholdvalue, decision 415 branches to “yes” branch 428 whereupon, at step 430,the compatibility list for the thread that just completed is checked. Ifthe thread that is currently running is already in the compatibilitylist, then the CPI for the thread is updated during step 430. In oneembodiment, the thread table keeps track of the last CPI, in which casethe latest CPI is inserted into the thread table in the field thatcorresponds to the identifier of the currently running thread. Inanother embodiment, the thread table keeps an average CPI value, inwhich case the newest CPI value is averaged in with the other valuesthat were achieved when the thread that just completed runs with thecurrently running thread. In addition, during step 430 a timestamp isrecorded to track the last time that the two threads ran together.

In the case where the currently running thread is not listed in the lastthread's compatibility list, a determination is made as to whether thereare any open slots (i.e., fields) in the compatibility list (decision435). If there is at least one open (i.e., currently unused) field,decision 435 branches to “yes” branch 438 whereupon, at step 440, thethread identifier of the currently running thread is recorded along withthe CPI value and a timestamp.

On the other hand, if there are no open slots in the compatibility listfor the thread that just completed, decision 435 branches to “no” branch442 which bypasses step 440 and performs another determination as towhether the CPI that was achieved between the two threads is better than(i.e., less than) the CPI of the poorest (i.e., highest) CPI currentlylisted in the compatibility list (decision 445). If the CPI achieved forthe two threads is better than one of the CPIs currently listed in thecompatibility list, decision 445 branches to “yes” branch 448 whereupon,at step 450, the thread identifier corresponding to the highest listedcompatible CPI is overwritten with the thread identifier of thecurrently running thread, the CPI value that was in the compatibilitylist is overwritten with the CPI that was just achieved, and the formertimestamp is overwritten with an updated timestamp reflecting the timeat which the CPI was achieved between the two threads.

If the CPI is not better than the poorest listed CPI in the threadtable, the compatibility list entries corresponding to the threadidentifier of the thread that just completed are left intact (i.e., notchanged) and decision 445 branches to “no” branch 452.

Similarly to steps 430 through 450 described above to update thecompatibility list for the thread that just completed executing, thesame steps are performed to update the compatibility list for thecurrently running thread. At step 460 the compatibility listcorresponding to the thread identifier that is currently running ischecked and, if the thread identifier of the thread that just completedis already listed, the data corresponding to the just completed threadis updated (i.e., the CPI and timestamp are updated). Again, in oneembodiment the last CPI is tracked while in another embodiment anaverage CPI is calculated and recorded.

In the case where the thread that just completed running is not listedin the currently running thread's compatibility list, a determination ismade as to whether there are any open slots in the compatibility list(decision 470). If there is at least one open (i.e., currently unused)field, decision 470 branches to “yes” branch 472 whereupon, at step 474,the thread identifier of the currently running thread is recorded alongwith the CPI value and a timestamp.

On the other hand, if there are no open slots in the compatibility listfor the currently running thread, decision 470 branches to “no” branch478 which bypasses step 475 and performs another determination as towhether the CPI that was achieved between the two threads is better than(i.e., less than) the CPI of the poorest (i.e., highest) CPI currentlylisted in the compatibility list (decision 480). If the CPI achieved forthe two threads is better than one of the CPIs currently listed in thecompatibility list, decision 480 branches to “yes” branch 485 whereupon,at step 490, the thread identifier corresponding to the highest listedcompatible CPI is overwritten with the thread identifier of the threadthat just completed executing, the CPI value that was in thecompatibility list is overwritten with the CPI that was just achieved,and the former timestamp is overwritten with an updated timestampreflecting the time at which the CPI was achieved between the twothreads.

If the CPI is not better than the poorest listed CPI in the threadtable, the compatibility list entries corresponding to the threadidentifier of the thread that just completed are left intact (i.e., notchanged) with decision 480 branching to “no” branch 492 bypassing step490.

Processing performed to update the threads' compatibility liststhereafter ends at 495.

FIG. 5 is a flowchart showing the steps taken to remove entries from athread's compatibility list. This procedure is called when the CPIachieved when two threads were executing at the same time on an SMTprocessor was worse than (i.e., higher than) a threshold set for thesystem (see FIG. 4, predefined process 420, that calls the processingshown in FIG. 5).

FIG. 5 processing commences at 500 whereupon, at step 510, thecompatibility list corresponding to the thread that just completedexecuting is read in order to determine whether the thread identifierfor the currently executing thread is listed as being a compatiblethread. In one embodiment, the compatibility list is stored in threadcontrol block 540. A determination is made as to whether the identifierof the currently running thread is listed in the last thread'scompatibility list (decision 520). If the current thread is listed inthe last thread's compatibility list, decision 520 branches to “yes”branch 525 whereupon, at step 530, data regarding the currently runningthread is removed from the compatibility list. In one embodiment, thecompatibility list data is stored in thread control block 540. On theother hand, if data pertaining to the currently running thread is notlisted in the compatibility list of the thread that just completed,decision 520 branches to “no” branch 535 bypassing step 530.

At step 550, the compatibility list corresponding to the currentlyrunning thread is read in order to determine whether the threadidentifier of the thread that just completed executing is listed asbeing a compatible thread. A determination is made as to whether thethread identifier of the thread that just completed executing is listedin the currently running thread's compatibility list (decision 560). Ifthe thread identifier of the thread that just completed executing islisted in the currently running thread's compatibility list, decision560 branches to “yes” branch 570 whereupon, at step 580, data regardingthe thread that just completed executing is removed from thecompatibility list. On the other hand, if data pertaining to the threadthat just completed executing is not listed in the compatibility list ofthe currently running thread, decision 560 branches to “no” branch 590bypassing step 580. Processing thereafter ends at 595.

FIG. 6 is a flowchart showing the steps taken to periodically clean upthe compatibility lists found in the thread control block. Processingcommences at 600 whereupon, at step 605, processing wakes up at periodicintervals, for example every two seconds.

Processing continues until the system is shutdown. Consequently, adetermination is made as to whether the system is being shutdown(decision 610). When the system is being shutdown, decision 610 branchesto “yes” branch 612 whereupon processing ends at 615.

On the other hand, if the system is not being shutdown, decision 610branches to “no” branch 618 to perform the thread clean up operations.At step 620 processing retrieves the current system time (timestamp). Astale timestamp value is calculated based upon the current time bysubtracting a stale time from the timestamp value (step 625). Forexample, an otherwise compatible thread that has not had its timestampvalue updated in the past two seconds may be considered “stale” and,therefore, no longer considered compatible with the thread. The reasonmay be because the other thread has terminated, the other thread hasbeen put to sleep waiting on another event to occur, or some otherreason that the other thread has not been scheduled to run along with anotherwise compatible thread.

At step 630, the first thread in the thread control block is read. Adetermination is made as to whether the thread control block dataincludes compatible thread information (decision 635). If the entry forthe thread includes compatible thread information, decision 635 branchesto “yes” branch 638 whereupon, at step 640, the timestamp correspondingto the first listed compatible thread is read. A determination is made,by comparing the timestamp to the calculated stale timestamp value, asto whether the thread listed in the compatibility list is stale andshould be removed from the list (decision 650). If the thread listed inthe compatibility list is stale, decision 650 branches to “yes” branch655 whereupon, at step 660, the stale thread is removed from thecompatible thread list. On the other hand, if the timestamp for thecompatible thread is within acceptable parameters (i.e., the thread isnot stale), then decision 650 branches to “no” branch 665 and the threadis kept in the compatible thread list.

A determination is made as to whether there are more threads listed inthe compatible thread list that need to be processed (decision 670). Ifthere are more threads listed, decision 670 branches to “yes” branch 672whereupon, at step 675, the timestamp for the next thread in thecompatible thread list is read and processing loops back to determinewhether the thread is stale and should be removed from the compatiblethread list. This looping continues until there are no more compatiblethreads listed for the thread read from the thread control block, atwhich point decision 670 branches to “no” branch 678.

A determination is made as to whether there are more threads listed inthe thread control block that need to be processed and have theircompatibility lists cleaned up (decision 680). If there are more threadsin the control block, decision 680 branches to “yes” branch 685whereupon, at step 690, the data for the next thread in the threadcontrol block is read and processing loops back to clean up any stalethreads listed in that thread's compatibility list.

This looping continues until all threads in the thread control blockhave been read, at which point decision 680 branches to “no” branch 695which loops back to step 605, causing processing to wait for the timeinterval to elapse before performing the clean up processing once again.Clean up processing continues until the system is shutdown, at whichpoint decision 610 branches to “yes” branch 612 and processing ends at615.

FIG. 7 is a diagram showing the scheduler swapping a poor performingthread from one SMT processor to another SMT processor in order toimprove overall system performance. Scheduler 700 reads data pertainingto threads that has been stored in thread control block 710 in order todetermine which threads should be swapped between processors in a systemwith multiple SMT processors.

The CPI data stored in the thread control block is used by the schedulerto identify poor performing threads from data gathered during thethreads' previous executions. In the embodiment shown in FIG. 7, two SMTprocessors are shown (760 and 785) each having two processor threads forexecuting two execution threads at the same time (processor threads 765and 770 corresponding to SMT processor 760 and processor threads 790 and795 corresponding to SMT processor 785). Each of the SMT processors havea run queue (run queue 755 corresponding to SMT processor 760 and runqueue 780 corresponding to SMT processor 785). The run queues identifythe threads that are scheduled to run on the processors. In the exampleshown, threads “aaa,” “ccc,” “eee,” and “ggg” are listed in run queue755 and, therefore, execute on SMT processor 760. Likewise, threads“bbb,” “ddd,” “fff,” and “hhh” are listed in run queue 780 and thereforeexecute on SMT processor 785.

Scheduler 700 determines which threads from the various run queues arethe poorest performing threads. Once the poor performing threads havebeen identified, process 705 within scheduler 700 swaps the threads fromone run queue to the other. In the example shown, thread “ggg” is thepoorest performing thread listed in run queue 755 while thread “hhh” isthe poorest performing thread listed in run queue 780. When process 705is performed, thread “ggg” will be placed in run queue 780 and thread“hhh” will be placed in run queue 755.

Because threads share processor resources in the SMT environment,swapping threads from one run queue to another puts the swapped threadin a pool of different threads with differing processor resources. Thegoal, therefore, of swapping threads is to find a more efficientenvironment for poor performing threads reducing contention forprocessor resources, thus improving thread efficiency. In addition, theswapping techniques shown in FIGS. 7-11 can be used in conjunction withthe SMT scheduling techniques shown in FIGS. 1-6 so that threads withina run queue are scheduled with more compatible threads within the samerun queue.

FIG. 8 is a flowchart showing the steps taken to update a thread's CPI.Processing commences at 800 whereupon, at step 810, a thread isdispatched by the scheduler to one of the processor threads that areincluded with the SMT processor. At step 820, an initial CPI value isretrieved from the processor. In one embodiment, the processor recordsthe number of cycles that were performed as well as the number ofinstructions that were executed. CPI is then computed as the number ofcycles divided by the number of executed instructions. The threadexecutes, at step 830, for some amount of time until the thread finishesits processing or is tasked out (i.e., timed out). When the thread isabout to finish executing, a notice is received, at step 840, informingthe process that the thread is about to finish processing. If theprocessing shown in FIG. 8 is being performed by the scheduler, then theprocess would determine that the thread is about to finish because thescheduler determines when threads are dispatched and tasked out. On theother hand, if the processing shown in FIG. 8 is performed by a processseparate from the scheduler, then the scheduler sends the process asignal when the thread is about to finish executing.

At step 850, the final CPI for the thread that just completed executingis retrieved. The CPI value is determined for the threads latest runcycle by computing the number of cycles that transpired while the threadwas executing as well as the number of instructions that were performedwhile during the thread's execution. [IS THIS THE NUMBER OF INSTRUCTIONSTHAT THIS THREAD COMPLETED, OR IS THIS THE NUMBER OF INSTRUCTIONS,OVERALL, THAT THE SMT PROCESSOR WAS ABLE TO COMPLETE?] The thread'slatest CPI is stored, at step 860, in thread table 870 (i.e., the threadcontrol block). At step 880, the thread's average CPI is computed byaveraging the CPI values stored in the thread table for this thread. Thethread's average CPI is then stored, at step 890, in thread table 870.

The processing shown in FIG. 8 is performed for each thread that isdispatched by the scheduler. For illustrative purposes, FIG. 8 shows theCPI tracking that is performed for a single thread. Because SMTprocessors operate on multiple threads simultaneously, the processingshown in FIG. 8 will be invoked multiple times in order to keep track ofthe various processor threads. For example, if the SMT processorprocesses two processor threads simultaneously, then the processingshown in FIG. 8 would either be executed twice (once for each thread) ormodified to track the CPIs of both threads.

FIG. 9 is a flowchart showing the steps taken to swap poor performingthreads between SMT processors. Processing commences at 900 whereupon,at step 905 the swap list (970) is initialized (i.e., cleared). Runqueue 915 corresponding to the first processor is selected at 910. Atstep 920, the worst performing thread from the first run queue isidentified by searching through thread table 925 for the thread in theselected run queue with the worst (i.e., highest) CPI.

A determination is made as to whether the CPI of the worst performingthread is above (i.e., worse than) a predetermined threshold (decision930). This determination is made to ensure that only poor performingthreads are swapped, otherwise additional resources are being taken toswap threads with adequate performance. If the thread's CPI is not worsethan the threshold, decision 930 branches to “no” branch 932 whichbypasses steps taken to write the thread's data into a swap list forswapping between the SMT run queues. On the other hand, if the thread'sCPI is worse than the threshold, decision 930 branches to “yes” branch934 whereupon, at step 935, previously swapped threads list 940 is readto see if the worst performing thread was recently swapped. Previouslyswapped thread list includes data about threads that have been swapped.This data includes the identifiers of the swapped threads, the CPI ofthe threads at the time the threads were swapped, and timestampsindicating the time at which each of the threads were last swapped.

A determination is made as to whether the worst performing thread waspreviously swapped, as indicated by the thread's identifier being foundin the previously swapped list (decision 945). If the thread waspreviously swapped, then decision 945 branches to “yes” branch 948whereupon a determination is made as to whether the thread's CPI isworse after the swap or was worse before the swap (decision 950). If thethread's CPI has improved (i.e., is not worse) after it was swapped,then decision 950 branches to “no” branch 952 whereupon, at step 955,the next worst performing thread from the selected processor run queueis identified, and processing loops back to determine whether thisthread was previously swapped and whether the thread's performance hasimproved or degraded following the swap. Returning to decision 950, ifthe selected thread's CPI is worse after being swapped, decision 950branches to “yes” branch 956.

A determination is made as to whether to include the identified threadin the swap list (decision 965). This decision can be based on a varietyof factors, such as whether the thread's CPI is worse than a giventhreshold and, if the thread was previously swapped, how long ago theswap occurred. It may be decided to not swap threads that have beenswapped very recently to avoid swapping the same threads back and forthamongst the processor run queues. If the thread is to be included in theswap list, decision 960 branches to “yes” branch 962 whereupon, at step965, swap list 970 is updated by writing the thread's identifier intothe list. On the other hand, if the thread is not to be included in theswap list, decision 960 branches to “no” branch 968 bypassing step 965.

A determination is made as to whether there are additional SMTprocessors from which to identify poor performing threads (decision975). In order to swap threads amongst processors, at least two SMTprocessors would be present in the computer system, so decision 975would branch to “yes” branch 978 at least once.

If there are more SMT processors to process, decision 975 branches to“yes” branch 978 whereupon, at step 980, the next processor in themultiprocessor system is selected and processing loops back to identifya poor performing thread from the selected SMT processor. This loopingcontinues until all SMT processors have been processed, at which pointdecision 975 branches to “no” branch 982 whereupon the threads in theswap list are swapped between processors (predefined process 985, seeFIG. 11 and corresponding text for processing details). At step 990,previously swapped threads list 940 is updated to record the threadidentifiers, latest CPI, and timestamps of the threads that are swapped.In addition, if more than two SMT processors are included in the system,then the previously swapped thread list also tracks the processor fromwhich the thread was taken. Processing thereafter ends at 995.

FIG. 10 is a flowchart showing the steps taken to identify poorperforming threads to swap in a multiple SMT processor system.Processing commences at 1000 whereupon, at step 1005, one of the threadsrunning on one of the SMT processors completes (i.e., is preempted, timesliced, etc.). At step 1010 the recently-completed thread's CPI is readfrom thread table 1015 that includes CPI related information about thethreads that are currently running.

A determination is made as to whether the recently-completed thread'sCPI is worse than a predetermined threshold (decision 1020). Thepredetermined threshold is a tunable value. Setting the threshold valuehigh will reduce the number of identified poor performing threads,while, conversely, setting the value low will increase the number ofidentified poor performing threads. If the recently-completed thread'sCPI is not worse than the threshold value, decision 1020 branches to“no” branch 1022 whereupon processing ends at 1025. On the other hand,if the recently-completed thread's CPI is worse than the thresholdvalue, decision 1020 branches to “yes” branch 1028 to perform furtherprocessing in order to ultimately determine whether therecently-completed thread should be swapped to another processor.

At step 1030, previously-swapped thread data structure 1040 is read.This data structure contains information about threads that werepreviously swapped from one SMT processor to another and includesinformation such as the thread's identifier, the thread's CPI at thetime at the time it was last swapped, and a timestamp indicating thelast time the thread was swapped.

A determination is made as to whether the recently-completed thread wasrecently swapped (decision 1050). If the recently-completed thread waspreviously swapped, decision 1050 branches to “yes” branch 1055whereupon a determination is made as to whether the recently-completedthread's CPI is worse after being swapped (decision 1060). If therecently-completed thread's CPI is not worse (i.e., it is the same orhas improved) after being swapped, decision 1060 branches to “no” branch1065 whereupon, at step 1070, the next worse performing thread isidentified on the selected processor and processing loops back todetermine if the newly identified thread is worse than the threshold,has been previously swapped, and whether the newly-identified thread'sCPI is worse after being swapped. This looping continues until eitherthe CPI of the identified threads (based upon the threads° CPIs) isbetter than the given threshold (at which point processing ends at1025), or until a thread with a CPI worse than the threshold isidentified that either has not been previously swapped (decision 1050branching to “no” branch 1075) or has a worse CPI after being swapped(decision 1060 branching to “yes” branch 1078).

When a poor performing thread has been identified, a determination ismade as to whether to include the identified thread in the swap list(decision 1080). This decision may be based on a variety of otherfactors, such as how recently the thread was previously swapped, howmuch better the thread performed on a different processor (i.e., amarginal improvement on a different processor may weigh against swappingthe thread), and the like. If the determination is to still include thethread in the swap list, decision 1080 branches to “yes” branch 1082whereupon, at step 1085, the swap list is updated by writing thethread's identifier to swap list 1090. On the other hand, if thedetermination is to not include the thread in the swap list, decision1080 branches to “no” branch 1092 bypassing step 1085. Processingthereafter ends at 1095.

FIG. 11 is a flowchart showing the steps taken to swap poor performingthreads between SMT processors. Processing commences at 1100 whereupon,at step 1110, identifiers for two threads are retrieved from the swaplist. In one embodiment, the swap list is used to swap threads from morethan two SMT processors. The swap list therefore identifies both thethreads and the SMT processor on which the thread last executed. Inaddition, the swap list may identify the processor to which the threadshould be scheduled, based upon the thread's past performance ondifferent processors. For example, in a system with four SMT processorswhere a poor performing thread has already been tried on the first,second, and third SMT processors, the swap list may indicate that thethread should be scheduled to run on the fourth SMT processor.

At step 1125, the first thread read from the swap list is placed on adifferent run queue (i.e., a run queue corresponding to a different SMTprocessor). At step 1130, the second thread read from the swap list isalso placed on a different run queue. In one embodiment, the firstthread is placed on the run queue corresponding to the second thread andthe second thread is placed on the run queue corresponding to the firstthread. At step 1140, thread table 1150 is updated reflecting thechanges made to the threads' run queues. At step 1160, data pertainingto these threads is written to previously swapped thread data structure1170. This data includes the threads' identifiers, the CPI of thethreads at the time they were swapped, and the current timestampindicating the time at which the threads were swapped.

At step 1175, the information corresponding to the threads that werejust swapped to different run queues is removed from swap list 1120. Adetermination is made as to whether there are more entries in the swaplist that need to be swapped (decision 1180). If there are additionalentries in the swap list, decision 1180 branches to “yes” branch 1190which loops back to swap the next two entries in the swap list (andremove the entries from the list). This looping continues until the swaplist is empty, at which point decision 1180 branches to “no” branch 1192and processing ends at 1195.

FIG. 12 illustrates information handling system 1201 which is asimplified example of a computer system capable of performing thecomputing operations described herein. Computer system 1201 includesprocessor 1200 which is coupled to host bus 1202. A level two (L2) cachememory 1204 is also coupled to host bus 1202. Host-to-PCI bridge 1206 iscoupled to main memory 1208, includes cache memory and main memorycontrol functions, and provides bus control to handle transfers amongPCI bus 1210, processor 1200, L2 cache 1204, main memory 1208, and hostbus 1202. Main memory 1208 is coupled to Host-to-PCI bridge 1206 as wellas host bus 1202. Devices used solely by host processor(s) 1200, such asLAN card 1230, are coupled to PCI bus 1210. Service Processor Interfaceand ISA Access Pass-through 1212 provides an interface between PCI bus1210 and PCI bus 1214. In this manner, PCI bus 1214 is insulated fromPCI bus 1210. Devices, such as flash memory 1218, are coupled to PCI bus1214. In one implementation, flash memory 1218 includes BIOS code thatincorporates the necessary processor executable code for a variety oflow-level system functions and system boot functions.

PCI bus 1214 provides an interface for a variety of devices that areshared by host processor(s) 1200 and Service Processor 1216 including,for example, flash memory 1218. PCI-to-ISA bridge 1235 provides buscontrol to handle transfers between PCI bus 1214 and ISA bus 1240,universal serial bus (USB) functionality 1245, power managementfunctionality 1255, and can include other functional elements not shown,such as a real-time clock (RTC), DMA control, interrupt support, andsystem management bus support. Nonvolatile RAM 1220 is attached to ISABus 1240. PCI-to-SCSI bridge 1280 provides bus control to handletransfers between PCI bus 1214 and SCSI bus 1285. SCSI device 1290 (i.e.a SCSI hard drive) communicates with other parts of computer system 1201using SCSI bus 1285.

Service Processor 1216 includes JTAG and I2C busses 1222 forcommunication with processor(s) 1200 during initialization steps.JTAG/I2C busses 1222 are also coupled to L2 cache 1204, Host-to-PCIbridge 1206, and main memory 1208 providing a communications pathbetween the processor, the Service Processor, the L2 cache, theHost-to-PCI bridge, and the main memory. Service Processor 1216 also hasaccess to system power resources for powering down information handlingdevice 1201.

Peripheral devices and input/output (I/O) devices can be attached tovarious interfaces (e.g., parallel interface 1262, serial interface1264, keyboard interface 1268, and mouse interface 1270 coupled to ISAbus 1240. Alternatively, many I/O devices can be accommodated by a superI/O controller (not shown) attached to ISA bus 1240.

In order to attach computer system 1201 to another computer system tocopy files over a network, LAN card 1230 is coupled to PCI bus 1210.Similarly, to connect computer system 1201 to an ISP to connect to theInternet using a telephone line connection, modem 1275 is connected toserial port 1264 and PCI-to-ISA Bridge 1235.

While the computer system described in FIG. 12 is capable of executingthe processes described herein, this computer system is simply oneexample of a computer system. Those skilled in the art will appreciatethat many other computer system designs are capable of performing theprocesses described herein.

One of the preferred implementations of the invention is an application,namely, a set of instructions (program code) in a code module which may,for example, be resident in the random access memory of the computer.Until required by the computer, the set of instructions may be stored inanother computer memory, for example, on a hard disk drive, or inremovable storage such as an optical disk (for eventual use in a CD ROM)or floppy disk (for eventual use in a floppy disk drive), or downloadedvia the Internet or other computer network. Thus, the present inventionmay be implemented as a computer program product for use in a computer.In addition, although the various methods described are convenientlyimplemented in a general purpose computer selectively activated orreconfigured by software, one of ordinary skill in the art would alsorecognize that such methods may be carried out in hardware, in firmware,or in more specialized apparatus constructed to perform the requiredmethod steps.

While particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein, changes and modifications may be madewithout departing from this invention and its broader aspects and,therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this invention. Furthermore, it is to be understood that theinvention is solely defined by the appended claims. It will beunderstood by those with skill in the art that if a specific number ofan introduced claim element is intended, such intent will be explicitlyrecited in the claim, and in the absence of such recitation no suchlimitation is present. For a non-limiting example, as an aid tounderstanding, the following appended claims contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimelements. However, the use of such phrases should not be construed toimply that the introduction of a claim element by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim element to inventions containing only one such element,even when the same claim includes the introductory phrases “one or more”or “at least one” and indefinite articles such as “a” or “an”; the sameholds true for the use in the claims of definite articles.

1. A computer-implemented method of scheduling threads for a pluralityof SMT processors, said method comprising: determining that a firstthread in a first run queue is a poor performing thread, wherein thefirst run queue corresponds to a first SMT processor; in response to thedetermination: writing a first identifier corresponding to the firstthread to a second run queue, wherein the second run queue correspondsto a second SMT processor; and removing the first identifier from thefirst run queue.
 2. The method as described in claim 1 furthercomprising: determining that a second thread in the second run queue isanother poor performing thread; in response to the determinationregarding the second thread: writing a second identifier correspondingto the second thread to the first run queue; and removing the secondidentifier from the second run queue.
 3. The method as described inclaim 1, wherein the determination further comprises: executing aplurality of threads listed in the first run queue, including the firstthread, on the first SMT processor, the executing further including:retrieving a number of cycles value for each thread indicating thenumber of cycles that occurred while each thread was executing;retrieving a number of instructions value for each thread indicating thenumber of instructions that were executed while each thread wasexecuting; dividing each number of cycles value by its correspondingnumber of instructions value, the dividing resulting in the CPI value;and recording the CPI value for each thread in the first queue; andidentifying the first thread as having a CPI value worse than aplurality of the other threads listed in the first run queue.
 4. Themethod as described in claim 3 further comprising: determining whethereach thread was previously moved from one of a plurality of SMTprocessors' run queues to the first SMT processor's run queue.
 5. Themethod as described in claim 4 further comprising: determining that theCPI of the first thread has degraded since being moved to the first SMTprocessor's run queue.
 6. The method as described in claim 3 furthercomprising: recording the first thread's identifier, the first thread'sCPI, and a timestamp to a previously swapped data structure.
 7. Themethod as described in claim 6 wherein the timestamp corresponds to atime selected from the group consisting of a time that the firstthread's CPI was calculated, and a time that the first thread'sidentifier was moved from the first run queue to the second run queue.8. The method as described in claim 3 further comprising: averaging theCPI value for each of the plurality of threads with one or more previousCPI values previously calculated for each of the threads, wherein therecorded CPI value includes the average CPI value for each thread. 9.The method as described in claim 1 further comprising: skipping one ormore worse performing threads in comparison to the first thread inresponse to determining that each of the worse performing threads haspreviously been moved to the first SMT processor's run queue and eachhas improved in performance since being moved; and identifying the firstthread after skipping the worse performing threads.
 10. The method asdescribed in claim 1 further comprising: sensing that the first threadis about to complete; computing a CPI value for the first thread uponits completion; determining that the CPI value is worse than a thresholdvalue; in response to the determination, checking whether the firstthread was previously moved to the first SMT processor from a previousSMT processor selected from a plurality of SMT processors that includesthe first SMT processor and the second SMT processor; in response todetermining that the first thread was previously moved, determining thatthe first thread's CPI is worse on the first SMT processor than it wason the previous SMT processor; and wherein the writing and the removingsteps are performed in response to determining that the first thread'sCPI is worse on the first SMT processor than on the previous SMTprocessor.
 11. An information handling system comprising: a plurality ofSMT processors; a memory accessible by the processors; a plurality ofsoftware threads and a plurality of run queues stored in the memory,wherein each of the run queues corresponds to one of the SMT processors;a thread scheduling tool for scheduling threads among the plurality ofSMT processors, the thread scheduling tool comprising software codeeffective to: determine that a first thread from the plurality ofsoftware threads that is in a first run queue selected from theplurality of run queues is a poor performing thread, wherein the firstrun queue corresponds to a first SMT processor selected from theplurality of SMT processors; in response to the determination, thesoftware code is effective to: write a first identifier corresponding tothe first thread to a second run queue, wherein the second run queuecorresponds to a second SMT processor selected from the plurality of SMTprocessors; and remove the first identifier from the first run queue.12. The information handling system as described in claim 11 wherein thesoftware code is further effective to: determine that a second thread inthe second run queue is another poor performing thread; in response tothe determination regarding the second thread: write a second identifiercorresponding to the second thread to the first run queue; and removethe second identifier from the second run queue.
 13. The informationhandling system as described in claim 11, wherein the determinationfurther comprises software code effective to: execute a plurality ofthreads listed in the first run queue, including the first thread, onthe first SMT processor, the execution comprising software codeeffective to: retrieve a number of cycles value for each threadindicating the number of cycles that occurred while each thread wasexecuting; retrieve a number of instructions value for each threadindicating the number of instructions that were executed while eachthread was executing; divide each number of cycles value by itscorresponding number of instructions value, the division resulting inthe CPI value; and record the CPI value for each thread in the firstqueue; and identify the first thread as having a CPI value worse than aplurality of the other threads listed in the first run queue.
 14. Theinformation handling system as described in claim 13 wherein thesoftware code is further effective to: determine whether each thread waspreviously moved from one of a plurality of SMT processors' run queuesto the first SMT processor's run queue.
 15. The information handlingsystem as described in claim 14 wherein the software code is furthereffective to: determine that the CPI of the first thread has degradedsince being moved to the first SMT processor's run queue.
 16. Theinformation handling system as described in claim 13 wherein thesoftware code is further effective to: record the first thread'sidentifier, the first thread's CPI, and a timestamp to a previouslyswapped data structure.
 17. The information handling system as describedin claim 16 wherein the timestamp corresponds to a time selected fromthe group consisting of a time that the first thread's CPI wascalculated, and a time that the first thread's identifier was moved fromthe first run queue to the second run queue.
 18. The informationhandling system as described in claim 13 wherein the software code isfurther effective to: average the CPI value for each of the plurality ofthreads with one or more previous CPI values previously calculated foreach of the threads, wherein the recorded CPI value includes the averageCPI value for each thread.
 19. The information handling system asdescribed in claim 10 wherein the software code is further effective to:skip one or more worse performing threads in comparison to the firstthread in response to determining that each of the worse performingthreads has previously been moved to the first SMT processor's run queueand each has improved in performance since being moved; and identify thefirst thread after skipping the worse performing threads.
 20. Theinformation handling system as described in claim 11 wherein thesoftware code is further effective to: sense that the first thread isabout to complete; compute a CPI value for the first thread upon itscompletion; determine that the CPI value is worse than a thresholdvalue; in response to the determination, check whether the first threadwas previously moved to the first SMT processor from a previous SMTprocessor selected from the plurality of SMT processors that includesthe first SMT processor and the second SMT processor; in response todetermining that the first thread was previously moved, determine thatthe first thread's CPI is worse on the first SMT processor than it wason the previous SMT processor, wherein the software code to write andremove are performed in response to determining that the first thread'sCPI is worse on the first SMT processor than on the previous SMTprocessor.
 21. A computer program product stored on a computer operablemedia for scheduling threads for a plurality of SMT processors, saidcomputer program product comprising: means for determining that a firstthread in a first run queue is a poor performing thread, wherein thefirst run queue corresponds to a first SMT processor; in response to thedetermination: means for writing a first identifier corresponding to thefirst thread to a second run queue, wherein the second run queuecorresponds to a second SMT processor; and means for removing the firstidentifier from the first run queue.
 22. The computer program product asdescribed in claim 21 further comprising: means for determining that asecond thread in the second run queue is another poor performing thread;in response to the determination regarding the second thread: means forwriting a second identifier corresponding to the second thread to thefirst run queue; and means for removing the second identifier from thesecond run queue.
 23. The computer program product as described in claim21, wherein the means for determining further comprises: means forexecuting a plurality of threads listed in the first run queue,including the first thread, on the first SMT processor, the means forexecuting further including: means for retrieving a number of cyclesvalue for each thread indicating the number of cycles that occurredwhile each thread was executing; means for retrieving a number ofinstructions value for each thread indicating the number of instructionsthat were executed while each thread was executing; means for dividingeach number of cycles value by its corresponding number of instructionsvalue, the dividing resulting in the CPI value; and means for recordingthe CPI value for each thread in the first queue, and means foridentifying the first thread as having a CPI value worse than aplurality of the other threads listed in the first run queue.
 24. Thecomputer program product as described in claim 23 further comprising:means for determining whether each thread was previously moved from oneof a plurality of SMT processors' run queues to the first SMTprocessor's run queue.
 25. The computer program product as described inclaim 24 further comprising: means for determining that the CPI of thefirst thread has degraded since being moved to the first SMT processor'srun queue.
 26. The computer program product as described in claim 23further comprising: means for recording the first thread's identifier,the first thread's CPI, and a timestamp to a previously swapped datastructure.
 27. The computer program product as described in claim 26wherein the timestamp corresponds to a time selected from the groupconsisting of a time that the first thread's CPI was calculated, and atime that the first thread's identifier was moved from the first runqueue to the second run queue.
 28. The computer program product asdescribed in claim 23 further comprising: means for averaging the CPIvalue for each of the plurality of threads with one or more previous CPIvalues previously calculated for each of the threads, wherein therecorded CPI value includes the average CPI value for each thread. 29.The computer program product as described in claim 21 furthercomprising: means for skipping one or more worse performing threads incomparison to the first thread in response to determining that each ofthe worse performing threads has previously been moved to the first SMTprocessor's run queue and each has improved in performance since beingmoved; and means for identifying the first thread after skipping theworse performing threads.
 30. The computer program product as describedin claim 21 further comprising: means for sensing that the first threadis about to complete; means for computing a CPI value for the firstthread upon its completion; means for determining that the CPI value isworse than a threshold value; in response to the determination that theCPI value is worse, means for checking whether the first thread waspreviously moved to the first SMT processor from a previous SMTprocessor selected from a plurality of SMT processors that includes thefirst SMT processor and the second SMT processor; in response todetermining that the first thread was previously moved, means fordetermining that the first thread's CPI is worse on the first SMTprocessor than it was on the previous SMT processor; and wherein themeans for writing and the means for removing are performed in responseto determining that the first thread's CPI is worse on the first SMTprocessor than on the previous SMT processor.